Provable Super-Convergence With a Large Cyclical Learning Rate

Conventional wisdom dictates that learning rate should be in the stable regime so gradient-based algorithms don't blow up. This letter introduces a simple scenario where an unstably large scheme leads to super fast convergence, with convergence depending only logarithmically on condition number of problem. Our uses Cyclical Learning Rate we periodically take one unstable step and several small steps compensate for instability. These findings also help explain empirical observations [Smith Topin, 2019] they show CLR maximum can dramatically accelerate lead so-called “super-convergence”. We prove our excels problems Hessian exhibits bimodal spectrum eigenvalues grouped into two clusters (small large). The is key enabling over eigen-spectrum.